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Current accountability systems fail to take into account what the public really wants 


from schools. 


By Derek Gottlieb and Jack Schneider 


Educational accountability has been relentlessly criticized 
for its imperfections over the past 15 years. These criticisms 
generally take aim at the narrowness of measurement systems, 
which rely chiefly on standardized test scores, as well as at 
the punitive sanctions associated with underperformance. 
Without a doubt, these are serious matters. But relatively little 
concern has been directed at a related problem: the failure of 
accountability systems to meaningfully engage the public. 

The problem is not that public outreach is entirely ab- 
sent. After all, states publish school-performance data and 
summative ratings of schools and districts for the explicit 
purpose of informing the public about how well their 
schools are doing. This, however, is an impoverished way 





of imagining public accountability. These systems ignore 
the actual interests and concerns of the public, focusing in- 
stead on a limited set of instrumental aims, such as raising 
student test scores. They sidestep the need for deliberation, 
instead employing formula-driven ranking and rating 
methods. And they do little to build capacity for planning 
and action, focusing instead on sanction and control. In 
sum, existing state accountability systems are public only 
in the most superficial sense. 

Putting the public back into public accountability sys- 
tems won't be easy. But we believe that truly democratic 
systems, though perhaps less efficient, represent our best 
hope for successful and sustainable school improvement. 
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The present system: Three shortcomings 


As policy rhetoric and existing measurement systems 
indicate, the state’s chief interest in educational account- 
ability is instrumental in nature. That is, to the state, good 
schools matter insofar as they produce industrious workers 
and competent citizens. But for members of the public, 
schools also serve a number of noninstrumental purposes 
— purposes worth pursuing for their own sake. Advocacy 
for arts education, for instance, or for unstructured play 
is rooted in the belief that such things matter, even if they 
produce no tangible or predictable returns to the state. 
Similarly, parents tend to support smaller classes whether 
or not they have a measurable impact on learning. That’s 
because they value the nature of the learning experience as 
much as the outcome. Additionally, while the state is uni- 
tary in its instrumental aims, the public is plural. Parents 
care about school quality for different reasons than other 
adults in a school’s community do. And different school 
communities, embedded as they are in different local 
ecologies, vary in their priorities and concerns. Existing 
accountability systems fall short in advancing the nonin- 
strumental and highly plural aims of the American public 
(for more, see Schneider, 2017). 

A second shortcoming in present accountability systems 
is the fact that states can’t “see” (Scott, 1998). Unlike mem- 
bers of the public, who gauge school quality holistically, 
states rely on standard, highly simplified information 
that can be compiled and combined. This results from the 
inherent challenge in trying to see and judge from afar, but 
it is exacerbated by the state’s unitary and instrumental 
aims, which lead the state to focus on a narrow range of 
performance metrics. Thus, because measurement systems 
tend to include only the data in which states are interested 
— test scores, graduation rates, and the like — they over- 
look any additional aims that schools serve. Consequently, 
states are blind to the actual strengths and weaknesses of 
schools, at least as they are valued by stakeholders. If the 
state can’t see schools from multiple perspectives, how can 
it know their holistic quality? 

A third shortcoming in present systems is their inflexi- 
bility, especially in prescribing consequences or solutions 
for lower-performing schools. This inflexibility, we argue, is 
directly related to the lack of local involvement in holding 
schools accountable. Working with limited information to 
advance a narrow set of aims, the state is ill-equipped to 
respond appropriately to the various factors shaping school 
performance. Additionally, state offices of education have 
limited staffing, which restricts their capacity for differen- 
tiated involvement. Further, states have a mandate to act 
impartially and consistently toward localities, which tends 
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to manifest in generic, one-size-fits-all policies. Because of 
all this, state interventions can look glaringly mistaken or 
misguided to local constituents. Putative “fixes” for under- 
performing schools — such as firing teachers, converting 
schools to charters, or closing schools entirely — are the 
equivalent of prescribing a broad-spectrum antibiotic to 
address all manner of symptoms, whatever the differences 
in the underlying illnesses. Through the lenses the state 
uses, schools with records of underperformance are all 
functionally the same, and they are dealt with in the same 
limited set of ways. 


Key features of a truly public system 


For an accountability system to truly serve the public, it 
must first recognize the plural aims of education, many of 
which are noninstrumental. In other words, it must answer 
the questions that stakeholders actually have about their 
schools. Do students feel safe and cared for? Is school pro- 
gramming rich and diverse? Are young people developing 
valued character traits and civic dispositions? 

To date, policy leaders have tried to address the short- 
comings of existing accountability systems by adopting a 
variety of additional metrics, but these often fail to reflect 
noninstrumental or plural aims. Consider how states have 
updated their accountability frameworks under the Every 
Student Succeeds Act (ESSA). In response to the mandate 
to include at least one additional metric, most have opted 
for student attendance on the grounds that it is a signif- 
cant predictor of outcomes like test scores and graduation 
rates. Instead of providing new kinds of information, 
then, they have offered new indicators for the same 
goals. Even laudable exceptions, like the CORE districts 
in California (https://coredistricts.org/our-data-research/ 
improvement-measures), generally treat measures like 
social-emotional learning as side dishes to the main course 
of test scores and graduation rates. 

If states seek to build truly public accountability systems, 
they must give stakeholders a voice in what a given school 
or district is accountable for — beyond instrumental metrics 
like academic proficiency and graduation rates. Presently, 
states often invite the public to participate in formal com- 
ment periods, but these usually only seek commentary 
about specific criteria employed in their otherwise inflexible 
systems. Accountability can and ought to be more demo- 
cratic than that. Whether through polling, focus groups, or 
civic deliberation, the public must have greater power over 
the way that school quality is conceptualized and tracked. 

In addition to enumerating further criteria by which 
schools will be judged, the public must also contribute to 
the actual evaluation of particular schools. Presently, most 


measurement and accountability systems rely on mecha- 
nistic rating systems, which generally rank schools against 
each other on a narrow set of measures. Done in the name 
of public accountability, such an approach falls short in 
two ways. First, it disregards the fact that communities may 
prize some aspects of school quality above others, meaning 
that plural public expectations will not conform to uniform 
state rating systems. Second, it ignores the fact that school 
quality is not a zero-sum game. All schools can succeed, 
even if some are stronger in particular areas than others, 
and the public must have a say in determining what consti- 
tutes success and failure. 

Public involvement in conducting evaluations would 
draw upon local knowledge as a resource rather than a hin- 
drance. Local stakeholders, who are attuned to their com- 
munity’s needs, are best positioned to apply what they see 
as the most salient criteria of school quality. Such an ap- 
proach would improve the substantive accuracy of school 
evaluations and ensure that the weaknesses and strengths 
of each school are fully accounted for on their own terms. 

Lastly, the public needs to have a say in charting the 
course of a given school in light of its annual evaluation. 
Given the breadth of the public’s interests in school quality, 
the results of an evaluation might imply a vast number of 
possible improvements. Which of those should be pursued, 
and in what order, cannot and should not be settled by 
market mechanisms or automatic state sanctions. Far from 
simply consuming and making personal choices based 
on test data, and then simply watching as the state purges 
an underperforming school’s faculty or shuts the school 
down entirely, the public ought to play a more active role in 
holding schools accountable for the aims those schools are 
meant to serve. 


What does this look like in practice? 


The first step in building a truly public accountability 
system is to engage stakeholders in defining a broad set 
of educational aims — aims more plural, and less instru- 
mental, than graduation rates or student standardized 
test scores. One model of such work can be seen in the 
Massachusetts Consortium for Innovative Education 
Assessment (MCIEA), where coauthor Jack Schneider 
serves as director of research. Through focus groups with 
hundreds of students, parents, teachers, principals, and 
community members, MCIEA created a School Quality 
Framework that consists of 16 constructs — from “school 
leadership” to “relationships” to “civic engagement.’ The 
framework is then used to guide measurement that cap- 
tures information aligned with what stakeholders value. 
Relying on member districts to provide administrative 





data, employing comprehensive surveys, and using 
performance assessment tools, the consortium provides 
stakeholders with rich information about student learning 
and school progress. Though MCIEA’s work is concentrated 
in seven Massachusetts districts, the research team is also 
supporting similar efforts elsewhere. 

The second step in building a truly public account- 
ability system is to engage stakeholders in conversations 
about what constitutes acceptable performance. Rather 
than ranking schools against each other and then auto- 
matically sanctioning the lowest quartile or quintile of 
performers, accountability systems should reflect the 
expectations and desires of the public. What is their 
yardstick for variables like “student sense of belonging” 
or “participation in creative and performing arts”? What 
are their standards when it comes to dimensions of 
school performance like “college-going and persistence” 
or “social and emotional health”? At MCIEA, stakeholders 
have played a significant role in shaping the performance 
benchmarks used to interpret each school’s data. The 
consortium plans to take such work a step further by 
bringing different stakeholder groups together to delib- 
erate in person. Empowered with broad and inclusive 
performance data, school communities will, we hope, 
meet regularly to discuss school quality and identify their 
schools’ strengths and weaknesses. 

Finally, in mapping out a future course for schools, 
algorithms should be tossed aside in favor of public con- 
venings. After all, if accountability is to be truly public, it 
must be authentically determined by those with a stake 
in schools, not by a unitary and mechanistic statewide 
system. MCIEA is outlining a model for such convenings, 
which would bring together community members, state 
officials, data analysts, school administrators, teachers, 
parents, and students. Having already established their 
schools’ strengths and weaknesses, and possessing a shared 
understanding of the community context, such groups 
would discuss and outline the road ahead for their schools 
or districts. This kind of deliberative model would be 
time-consuming, certainly. And it would have to sidestep 
the bureaucratic nature of school board meetings, while 
avoiding the unrepresentative nature of voluntary groups. 
Additionally, it would require significant capacity building. 
Ultimately, however, it might also make school improve- 
ment efforts more likely to be accepted and to succeed. 


In response to potential objections 


Obvious objections to this sort of proposal may imme- 
diately arise. Won't allowing significant local control in 
accountability undercut the state’s ability to meet its re- 
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sponsibility for ensuring equity among schools throughout 
the state? Isn’t judging each school differently the very def- 
nition of unfairness? Won't localities be able to game the 

system and adjust criteria to make their schools look good? 

The possibility of local control undercutting state over- 
sight responsibilities is an important worry. But involving 
local stakeholders in making decisions does not exclude 
the state’s official concerns. The involvement of state of- 
ficials in these deliberative bodies means that the state’s 
interest in equity and quality is represented and, indeed, 
enforced. The goal here is not to omit or minimize the 
instrumental interests that the state serves; it is to include 
the various needs and concerns of local stakeholders, even 
when these are noninstrumental. 

As for the risks associated with treating schools dif- 
ferently, we have a substantially more robust version of 
fairness in mind than the sort expressed through present 
accountability systems. Current practices ensure fairness 
through a mechanical version of objectivity in which 
schools are evaluated through an algorithmic combination 
of a handful of measures, without significant public input 
or engagement. We believe that deliberations are fairer 
than algorithms and that a more substantive and less pro- 
cedural version of fairness will be both more just and more 
democratic. Because a school’s real quality depends on the 
values of its community, a school must be evaluated holisti- 
cally and multidimensionally. Putting local representatives 
in conversation with one another and with state officials 
maximizes the likelihood of doing justice to the quality of 
a particular school while still responding to the concerns of 
the state. 

With respect to the possibility of gaming the system, the 
very objection is based on a mistake. Gaming the system 
is only desirable in an environment where accountability 
is limited to doling out rewards and punishments on 
the basis of performance against a set of measures that 
the community does not necessarily care about. In other 
words, gaming the system is only a threat under current 
accountability systems — ones that privilege accountability 
to the state. But schools belong to the public they serve, 
and local people are the ones most likely to take an interest 
in the quality of their schools, just as they are most likely 
to take an interest in the quality of their roads or the safety 
of their neighborhoods. Under a genuinely public account- 
ability regime, there is no impersonal system to game. The 
state, instead, channels its efforts into ensuring that data 
are comprehensive and transparent and that the public has 
the support it needs in assessing school performance. 

Finally, some may object to how unwieldy this whole 
process may become. Such a concern is answerable in two 
ways. First, current practices are far from simple, despite 
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their widespread acceptance. Present accountability sys- 
tems require massive expenditures of money and time that 
might be spent more profitably in other ways. Though the 
wheels of accountability now churn along seemingly with- 
out effort, we must remember that these systems have con- 
sumed tremendous resources over the past two decades. A 
second response to the objection of unwieldiness is to reit- 
erate the local nature of these deliberative groups. In each 
case, we imagine cohorts of 10-20 people, some of whom 
— namely district and state officials — will be extremely 
practiced, serving on these groups multiple times per year. 
This process is only chaotic to the same extent that jury 
deliberations are chaotic. Such processes are enshrined in 
our public practices because they are internally related to 
the ideal of democratic governance. Deliberations would 
not be simple or easy, but the alternative is fundamentally 
unjust. 


Embracing democracy 


Our public schools operate within a pluralistic dem- 
ocratic society. In our national mythos, this has been a 
source of unique strength, even as it has presented chal- 
lenges. But present school accountability systems seem to 
wish away this complexity — obscuring the challenges of 
pluralistic democracy behind a system that will smooth 
over the vagaries of difference among us. Our collective de- 
lusion is that we can build an accountability machine, set 
it in motion, and let the unimpeachable results roll in, free 
from the responsibilities of situational judgment. This is an 
ignoble wish. While present accountability systems might 
appear to fulfill a responsibility to the public, they actually 
represent a negation of democracy. 

Can members of the public agree on a set of relevant 
educational aims? Can they take on the responsibility 
of assessing school quality? We believe so. Democratic 
processes are not fail-safe or foolproof, but mechanistic 
accountability systems are an inadequate alternative. If 
human judgment is imperfect or frail, the solution is not 
to eliminate the possibility of judgment, but rather to 
put diverse judgments together in serious conversation. 
Democratic justice has always required the judgments of 
one’s peers and of one’s community. And, though it may 
not come easy, the importance of educational accountabil- 
ity merits an equally high standard. « 
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